Method and apparatus for gathering and evaluating information

ABSTRACT

A continuous information gathering and evaluating system uses five highly coupled databases and software for gathering user input (respondent software), for evaluating input (evaluator software), and for administering the system (administrative software). The first database includes the categories of users for an organization. The second database stores information about each user. The third database includes all topics and issues of interest and queries for each combination of topic, issue, and respondent category. Responses to query statements are preferably collected using a 7 point Likert scale. The fourth database is the repository for the responses from users and appropriate statistics based on the responses. The fifth database includes authorization data and rules that determine how the process and system are implemented for a particular organization.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates to a method and apparatus for gathering andevaluating qualitative inputs or feedback, including perceptions andopinions, from employees and customers.

2. Discussion of the Background

Leaders and managers use both quantitative and qualitative informationto run their organization. Qualitative information from employees andcustomers is both difficult to gather and difficult to evaluate.However, qualitative information is often the most relevant input orfeedback that a leader or manager needs.

Methods for gathering qualitative information from employees andcustomers range from very ad hoc (e.g. discussion over a cup of coffee,suggestion boxes, chat rooms, etc.) to standardized surveys. The ad hocmethods lack the consistency that is needed to make them reliable andrepeatable. Each input must be evaluated separately—with no standardsfor comparison and no way to quantify the results. The value of mostinformation gathered in an ad hoc method is dependent on the skills ofthe leader or manager who happens to be listening or reading the input.Ad hoc methods often provide inputs from a very small, vocal group ofemployees or customers. When this occurs leaders and managers cannotdetermine if the perceptions and opinions reflect the views of amajority or just a few individuals. The ad hoc methods lack theefficiency that is essential to making them a consistently valuablesource of critical, qualitative information.

More standardized methods, such as surveys, provide the much-neededconsistency, but they are not efficient enough to make them a continuoussource of qualitative information that is needed by leaders andmanagers. For example, surveys take considerable time to develop,distribute, collect, and analyze. Once the analysis is complete theresults must be conveyed to the appropriate managers for action. All ofthis takes both effort (labor hours and funding) and time (actualcalendar days) that is not available. Time delays from the start ofsurvey development until the appropriate manager has resultssignificantly reduce the value of the information because it is toolate. That is, when done right, a survey takes months and theinformation is quite often no longer important to the leaders andmanagers, or the information is too late to help. In addition, standardsurveys cannot be used to identify trends in perceptions or opinionsunless the same questions, or cleverly worded questions that aresimilar, are asked on repetitive surveys. The repetition makes employeesand customers lose interest and surveys of this type are most likelyleft unanswered.

What is not available is a method and system that allows leaders andmanagers to rapidly identify and gather qualitative information fromemployees and customers on topics and issues that are important to themat the time the information is gathered. Existing methods and systems donot allow leaders and manager to identify trends in employee andcustomer perception in an effective and efficient manner. The advent ofcomputers and the Internet has provided some efficiency in thedistribution, collection and analysis of standardized surveys. However,these technologies have not overcome the major impediments that makestandard surveys ineffective for gathering and evaluating continuous,consistent and cost effective inputs from employees and customers. Belowis a list of the major steps in the development and use of standardsurveys:

Standard Survey Computer and Step Process Internet Impact  1 Determinethat perceptions and opinions of None employees and/or customers must becollected  2 Determine what specific perceptions and None opinions areneeded  3 Determine who should provide perceptions None and opinions  4Develop and approve a process or obtaining None perceptions and opinions 5 Develop a format for an instrument for None obtaining perceptions andopinions  6 Develop query statements or questions for None obtainingperceptions and opinions  7 Distribute the instrument for obtainingInternet impact is perceptions and opinions potentially significant  8Collect and load data from each respondent Significant impact ofcomputers on electronic or machine readable responses; Internet impacton collection is potentially significant  9 Analyze data Computer impactis significant on statistical computations; computer impact oninterpreting results is minimal unless the formats for presentation areconsistent 10 Develop and distribute reports Computer impact on standardreports is significant

Out of the ten steps listed above, the use of computers and the Internethas helped only in the collection and loading of the data (in the caseswhere the inputs are in electronic form or machine readable form), theanalysis of the data, and the development and distribution of standardreports. The use of computers and the Internet has not helped in themajority of the steps listed above.

What is needed is a way to overcome the time consuming process ofidentifying the information needed, translating the identifiedinformation into an instrument for gathering the information, selectingthe people who are to provide the information, compiling or modifyingdistribution lists to reflect the selection of respondents, analyzing(evaluating) the information and distributing the information to all thepeople who need it within an organization.

Specifically, what is needed is an effective and efficient method andsystem that:

Allows leaders and managers in an organization to obtain inputs fromemployees and customers on specific topics (products, activities ortasks, projects, programs, policies, benefits, etc.) in theirorganization at any time and with any frequency they desire;

Allows leaders and managers in an organization to obtain input fromemployees and customers on general issues (morale, adequacy ofcommunications, customer satisfaction, level of team work, etc.);

Allows leaders and managers to identify trends with respect to employeeand customer perceptions and opinions on both specific topics andissues;

Allows leaders and managers to identify when employees and customersperceive certain issues (e.g. inadequate communication) as veryimportant with respect to specific topics (e.g. health benefits);

Allows leaders and managers to identify which groups of respondents areproviding both positive and negative perceptions and opinions onspecific topics and issues;

Allows all managers and leaders in an organization to rapidly evaluateinputs to help them gain a common view of the perceptions and opinionsof employees and customers—thereby enhancing their ability to operate asa cohesive leadership team;

Provides leaders and managers at every level in the organization withcritical, qualitative information they need to correct problems and makethe organization more effective and efficient without having to wait forinformation and direction to flow down from a higher level leader ormanager;

Allows employees and customers to pick the topics that are mostimportant to them when they provide periodic input—thereby notsubjecting them to many questions on lengthy surveys that are notrelevant to them;

Ensures leaders and managers that the methods and techniques used togather employee and customer input is consistent and reliable—withoutthe normal time and effort needed to develop this confidence onindividual survey instruments.

SUMMARY OF THE INVENTION

The present invention addresses the problems of the prior art byproviding a method and system that effectively and efficiently providesleaders and managers with timely, valuable, qualitative inputs fromemployees, customers, and other users.

The present invention provides a process and system that allows leadersand managers in an organization to continuously select and update theimportance of topics and issues for which that they require perceptionsand opinions. From this initial list, the process and system provideseach user with a list of relevant topics, based on the respondentcategories the to which the user belongs. The leaders and managers canset the number of topics and the frequency that an employee may respond.The process and system provides a set of queries that elicits theopinions and perceptions of the user for each topic chosen by the user.In some embodiments, the user is required to respond to queries fortopics designated as mandatory by the leaders and managers.

The process and system provide leaders and managers with the capabilityto rapidly evaluate inputs from thousands of employees and customers.The process and system automatically alert leaders and managers to bothpositive and negative trends in employee and customer perceptions andopinions. The process and system provide leaders and managers with thecapability to analyze specific topics or issues and determine whichcategories of respondents are providing positive, negative or specificresponses.

The system uses five highly coupled databases and software for gatheringuser input (respondent software), for evaluating input (evaluatorsoftware), and for administering the system (administrative software).The first database includes the categories of users for an organization.The second database stores information about each user, including name,password (if password protection is desired in a particular system), andcategory. The third database includes all topics that are of interest tothe leaders and managers and all issues that are associated with thesetopics. The list is dynamic and grows based on the needs of the leadersand managers and free text comments received from the employees andcustomers. The third database also contains queries for each combinationof topic, issue, and respondent category. To ensure consistency andrapid evaluation, all query statements are preferably “Seven PointLikert Statements” that range from “Strongly Disagree” to “StronglyAgree” in preferred embodiments. The fourth database is the repositoryfor the responses from users and appropriate statistics based on theresponses. The dates of the responses and the category of therespondents are also stored. The fifth database includes authorizationdata and rules that determine how the process and system are implementedfor a particular organization. It is important to note that while thesedatabases are described as separate, the databases could also be partsof a single database.

The respondent software automatically links each employee with theappropriate topics for which they are supervisors, performers orcustomers. Respondents can then choose the topics that are of mostinterest to them. The evaluator software allows leaders and managers toreview all the topics for which they have permission. The softwareautomatically identifies trends in topics and issues and allows theevaluator to “drill down” in detail to identify the most relevant issuesfor a topic and which respondent categories are providing positive andnegative inputs. The process and system are preferably implemented on anorganization's Intranet or the Internet.

BRIEF DESCRIPTION OF THE DRAWINGS

The aforementioned advantages and features of the present invention willbe more readily understood with reference to the following detaileddescription and the accompanying drawings in which:

FIG. 1 is a block diagram of an information gathering system accordingto a preferred embodiment of the present invention.

FIG. 2 is a view of a login window of the system of FIG. 1.

FIG. 3 is a view of a registration window of the system of FIG. 1.

FIG. 4 is a view of a respondent category selection window of the systemof FIG. 1.

FIG. 5 is a view of a profile window of the system of FIG. 1.

FIG. 6 is a view of an activity selection window of the system of FIG.1.

FIG. 7 is a view of a hot topic specification window of the system ofFIG. 1.

FIG. 8 is a view of a topic selection window of the system of FIG. 1.

FIG. 9 is a view of a query window of the system of FIG. 1.

FIG. 10 is a view of a selection window of the system of FIG. 1.

FIG. 11 is a view of a summary window of the system of FIG. 1.

FIG. 12 is a view of an evaluation options window of the system of FIG.1.

FIG. 13 is a view of a participant category evaluation window of thesystem of FIG. 1.

FIG. 14 is a view of an issue evaluation window of the system of FIG. 1.

FIG. 15 is a view of a trend analysis window of the system of FIG. 1.

FIG. 16 is a view of a query analysis window of the system of FIG. 1.

FIG. 17 is a view of a trend summary window of the system of FIG. 1.

DETAILED DESCRIPTION

In the following detailed description, a plurality of specific details,such as specific issues and topics and types of communications networks,are provided in order to provide a through understanding of the presentinvention. The details discussed in connection with the preferredembodiments should not be understood to limit the present invention.

FIG. 1 illustrates an information gathering and evaluating system 100according to a preferred embodiment of the present invention. The system100 comprises a database server 110, which includes a server 120 and adatabase 130. The database server 110 is connected to a communicationsnetwork 140. In preferred embodiments, the communications network is theInternet or an organization's intranet. However, any communicationsnetwork can be used. The communications network may also comprise acombination of networks, such as a LAN for local users and the Internetfor remote users. Also connected to the communications network are aplurality of end user computers 150-152.

The database 130 includes five highly coupled databases. The firstdatabase includes the categories of users for an organization. Anexample of a portion of a category database for a school system isprovided in Table 1 below:

TABLE 1 List of User Categories and Subcategories for a School SystemSuperintendent Associate Superintendents Associate Superintendent ofInstruction Associate Superintendent of Management AssociateSuperintendent of School Services Area Associate Superintendents Area 1Associate Superintendent Area 2 Associate Superintendent Area 3Associate Superintendent Division Directors Director of Curriculum andStaff Development Director of Student Services Director of Planning andAssessment Director of Personnel Director of Finance Director of RiskMgt and Security Director of Information Services Director ofTransportation Director of Maintenance Staff Director of Food ServicesDivision Supervisors and Managers Curriculum Supervisors ScienceCurriculum Supervisor Math Curriculum Supervisor Social StudiesCurriculum Supervisor English Curriculum Supervisor Foreign LanguageCurriculum Supervisor Fine Arts Curriculum Supervisor Vocational andCareer Curriculum Supervisor Building Trades Curriculum SupervisorSupervisor of Title 1 Administrative Coordinator of Head StartAdministrative Coordinator of Instruction Support Team Supervisor ofGifted Education & Special Programs Supervisor of Alternative Education& Summer Schedule Supervisor of Adult Education Program Supervisor ofVocational and Career Education Supervisor of Instructional TechnologySupervisor of Multicultural Education Supervisor of Guidance SpecialEducation Supervisors PACE Principals Supervisor of Testing andAssessment Supervisor of Facilities & Real Property Planning Supervisorof Benefits Supervisor of Classified Personnel Supervisor of ElementaryPersonnel Supervisor of Secondary Personnel Director of Data ProcessingSupervisor of Budget Supervisor of Fiscal Operations Supervisor ofFinancial Services Construction Management Director of ConstructionProject Managers Supervisor of Media Production Supervisor of CommunityRelations Plant Operations Management Supervisor of Plant OperationsCustodial Managers Supervisor of Supply Services Supervisor ofPurchasing Division Staff Clerical Staff Adult Education StaffAlternative Education and Summer School Staff Title 1 Staff CurriculumStaff Gifted Education and Special Programs Staff Head Start StaffMulticultural Education Staff Organizational Staff Development TeamMembers Instructional Support Team Members Special Education StaffAdministrative Coordinator of Adult Education Administrative Coordinatorfor Guidance Student Services Staff Testing and Assessment StaffFacilities and Real Property Planning Staff Grants Coordinator(s)Benefits Staff Personnel Staff Budget Staff Finance Staff ConstructionStaff Media Production Staff Community Relations Staff Food ServicesStaff Transportation Staff Supply Services Staff Purchasing StaffMaintenance Staff Custodial Services Staff Training and Safety StaffVehicle Maintenance Staff School Administrators Principals Principals -Elementary Principals - Middle/Intermediate Principals - SecondaryAssistant Principals Assistant Principals (D) - Discipline AssistantPrincipals (S) - Science Assistant Principals (M) - Math AssistantPrincipals (E) - English Assistant Principals (SS) - Social StudiesAssistant Principals (LA) - Language Arts Assistant Principals (FA) -Fine Arts Assistant Principals (FL) - Foreign Language AssistantPrincipals (VC) - Vocational & Career Ed Assistant Principals (BT) -Building Trades Assistant Principals (SE) - Special Education AssistantPrincipals (H&PE) - Health and Physical Education Assistant Principals(1st) - 1st Grade Assistant Principals (2nd) - 2nd Grade AssistantPrincipals (3rd) - 3rd Grade Assistant Principals (4th) - 4th GradeAssistant Principals (5th) - 5th Grade Assistant Principals (6th) - 6thGrade Administrative Assistants Administrative Assistants (D) -Discipline Athletic Director Non-Academic School Staff Non-AcademicSchool Staff Managers Psychological Support Services Providers SchoolClerical Staff School Maintenance/Facility Staff Custodial Staff ManagerCustodial Staff Food Service Staff Security Staff Athletic ActivityStaff Medical Staff Library Staff Printed Media Film Library StaffAcademic Staff Academic Department Chairs Mathematics Department ChairEnglish Department Chair Science Department Chair Social StudiesDepartment Chair Language Arts Department Chair Fine Arts DepartmentChair Foreign Language Department Chair Vocational and Career EducationDepartment Chair Building Trades Department Chair Health and PEDepartment Chair Special Education Department Chair Teachers MathematicsTeachers English Teachers Science Teachers Social Studies TeachersLanguage Arts Teachers Fine Arts Teachers Foreign Language TeachersVocational and Career Education Teachers Building Trades Teachers Healthand PE Teachers Special Education Teachers 1st Grade Teachers 2nd GradeTeachers 3rd Grade Teachers 4th Grade Teachers 5th Grade Teachers 6thGrade Teachers Adult Education Teachers Multicultural Education TeachersHomebound Teachers Gifted and Talented Teachers Head Start TeachersSummer School Teachers Title 1 Teachers Alternative Education TeachersHead Guidance Counselor Guidance Counselors Extra-curricular ActivitiesSponsors Coaches Assistant Coaches Students Mathematics Students EnglishStudents Science Students Social Studies Students Language Arts StudentsFine Arts Students Foreign Language Students Vocational and CareerEducation Students Building Trades Students Health and PE StudentsSpecial Education Students 1st Grade Student 2nd Grade Student 3rd GradeStudent 4th Grade Student 5th Grade Student 6th Grade Student AdultEducation Students Multicultural Education Students Gifted and TalentedStudents Homebound Students Head Start Students Summer Students Title 1Students Alternative Education Students Athletes Extra-curriculaStudents Students Using Psychological Support Services VolunteersAthletic Booster Club Member Community Parents Mathematics ParentsScience Parents English Parents Social Studies Parents Language ArtsParents Fine Arts Parents Foreign Language Parents Vocational and CareerEducation Parents Building Trades Parents Special Education ParentsHealth and PE Students 1st Grade Parent 2nd Grade Parent 3rd GradeParent 4th Grade Parent 5th Grade Parent 6th Grade Parent EWSAdministrator RESA VII Trial User Groups Superintendents RESA BoardMembers Gear Up Site Coordinators Gear Up County Contacts StaffDevelopment Coordinators Staff Development Council Members PDS GroupStudent Teachers PDS Evaluator

The database includes three “levels” of user categories. The highest, ortop, level category includes categories such as School Administrators.The next level category includes subcategories such as Principals, whilethe third level category involves a still further subcategory such asElementary School Principals. Not all top level categories havesubcategories. For example, the Community group in Table 1 has nosubcategories. Inclusion in a low level category automatically resultsin inclusion in all upper level categories. Thus, a user who is part ofthe elementary school principal category is automatically a part of theprincipal and school administrator categories. This is not true in thereverse direction, however. Furthermore, it is possible to be part of acategory with out being a part of any of its subcategories. For example,it is possible for a user to be a part of the Volunteers categorywithout being part of its only subcategory, Athletic Booster ClubMembers. This would be the case for a user such as a user who volunteersto help younger children learn to read after school.

The selection of user categories and subcategories is made in light ofthe desired level of focus. As will be further discussed below, eachissue may have multiple queries, some of which may be generated forspecific categories or subcategories of users. The decision as to howfocused the respondent categories are will depend in part upon the needor desirability of tailoring questions for specific groups and in partupon the need or desirability of analyzing responses, even if theresponses are to general questions, among a specific set of respondents.For example, one issue applicable to a school environment is schoolsafety. A general question that may be asked of all respondents is “Ialways feel safe when I am in the school or on the grounds.” Althoughthe query may not be directed towards a specific group, it may be verydesirable to analyze the responses by user categories such as studentsand teachers, and subcategories such as high school students vs.elementary school students, etc. In some embodiments of the invention,it is possible to gather responses for certain issues from certaincategories and/or subcategories of respondents. Thus, if absenteeism isbelieved to be an issue only for high schools, then queries may bedirected at only of high school teachers and students, rather thanincluding all subcategories of teachers and students. In preferredembodiments of the invention, the respondent category database isdynamic; that is, categories and subcategories are added or deleted asperceived needs change.

The second database stores information about each user, including name,password (if password protection is desired in a particular system), andcategory. The categories may be freely selected by the users, or thecategories for each user may be predefined. This database may storeother information as well, such as addresses, identification numbers,etc.

The third database includes all topics that are of interest to theleaders and managers and all issues that are associated with thesetopics. A topic is some element of an organization that is of interestto leaders and managers. Topics can be an organizational process orfunctional area (e.g. clerical support), a specific program or project(e.g., a Head Start program in elementary schools), or any uniquecombination of activities in an organization. Topics are unique and donot overlap. Issues are areas within an organization that overlap one ormore topics. For example, “morale” is an issue that is of interestacross most functional areas and on most major programs and projects. Inpreferred embodiments, an issue cannot belong to a single topic; rather,it is a topic unto itself. The set of issues that are applicable to eachspecific topic varies among topics.

A partial list of topics and issues that may be of interest in a schoolsystem is presented in Table 2 below:

TABLE 2 Partial List of Topics and Issues of Interest in a School SystemTopic Issue instruction absenteeism class participation communicationcustomer satisfaction discipline effectiveness efficiency environmentevaluation morale parent involvement participation resources studentmotivation substitute teachers work load worker background ClericalSupport communication customer background customer satisfactionmanagement morale policies & procedures team work work load workerbackground General Achievement Testing appropriateness effectivenesspositive impacts negative impacts validity timing quantity

In preferred embodiments, each issue is applicable to more than onetopic. Said another way, if an area of interest that might be classifiedas an issue is applicable to only one topic, then that area of interestis made a topic unto itself, rather than being classified as an issuerelevant to a topic. The list of topics and issues is preferably dynamicand grows based on the needs of the leaders and managers and free textcomments received from the employees and customers.

The third database also contains queries for each combination of topic,issue, and respondent category. The query statements are preferablyeither very positive or very negative statements about the topic and itsrelated issues. An exemplary partial listing of queries for the GeneralAchievement Testing topic of Table 2 is presented below in Table 3:

TABLE 3 Sample Partial List of Query Statements for Elementary andSecondary Schools Respondent Query Topics Issues Category StatementsGeneral appro- teacher, State mandated tests to Achievement priatenessadministrators assess the quality of a Testing schools teaching are agood idea. I don't think state mandated tests to assess how well schoolsteach is a good idea. State mandated tests that are used to accreditschools help keep the quality of education high. I support mandatedtesting to ensure each school maintains high standards. effectivenessparents State and County mandated tests are valid measures of what mychild has learned. My experience indicates that State and Countymandated tests are good measures - they help identify what our childrenhave learned. State and County mandated tests are very useful ways toidentify what a I have learned at our school. State and County mandatedtests are not valid measures of what our children learned at our school.effectiveness students State and County mandated tests are validmeasures of what I have learned. My experience indicates that State andCounty mandated tests are good measures - they help identify what I havelearned. State and County mandated tests are very useful ways toidentify what a I have learned at our school. State and County mandatedtests are not valid measures of what I have learned at our school.positive teacher, I think state mandated tests impacts administratorsthat measure student learning help motivate our students. Students seestate mandated tests that will determine if they can move to the nextgrade and eventually graduate from high school as a real motivator.State mandated tests that determine if a student moves on or is heldback do not help motivate our students. State mandated tests thatmeasure what students have learned are a definite asset in our move toimprove the quality of our education system. I believe that statemandated tests that determine what our students are learning will helpus raise the quality of education at our school. State mandated teststhat attempt to determine what students have learned do not help improvethe quality of education at our school. Positive students I think statemandated impacts tests that measure learning help motivate me to learnmore. I see state mandated tests that determine if I can move to thenext grade and eventually graduate from high school as a real motivator.State mandated tests that determine if I move on or am held back do notmotivate me to learn. State mandated tests that measure what I havelearned are a definite asset in our school's move to improve the qualityof our education system. I believe that state mandated tests thatdetermine what I am learning will help raise the quality of education atour school. State mandated tests that attempt to determine what I havelearned do not help improve the quality of education at our school.

The fourth database is the repository for the responses from users. Inorder to facilitate analysis, responses are preferably gatheredaccording to a uniform scale oriented along the same direction. Inhighly preferred embodiments, the responses are gathered in the form of7 point Likert scale. The reliability of a 7 point Likert scale is wellestablished. As discussed above, the queries are positive or negativestatements about the topic/issue. Respondents use the scale to recordtheir agreement or disagreement with the query statements. Therefore,when a respondent “strongly agrees,” “agrees,” or “mildly agrees,” witha very positive statement (a 7, 6, or 5, respectively, on a 7 pointLikert scale), the response is placed in the positive set of responses.Likewise, when the respondent “strongly disagrees,” “disagrees,” or“mildly disagrees” (a 1, 2, or 3, respectively, on a 7 point Likertscale), the response is placed in the negative set of responses. (A 4 onthe Likert scale corresponds to no opinion.) The placement of responsesis reversed when the query statement is very negative. That is,disagreeing with a negative statement is viewed as a positive perceptionor opinion.

In some embodiments, the raw responses are stored. That is, a separateentry in the database is made indicating the respondent category, thedate of the response, the query and the response itself. Those of skillin the art will appreciate that the storage requirements for such ascheme can be impractically large. Therefore, in embodiments with largenumbers of users, statistical information rather than raw numbers, maybe stored. Statistical information includes information such as thepercentages of positive and negative responses and the total number ofresponses for each query statement, along with an indication of whichrespondent category the respondents belong to and the date of theresponse. In such embodiments, the statistical information may begenerated in a batch process scheduled to run overnight.

The fifth database includes authorization data and rules that determinehow the process and system are implemented for a particularorganization. The rules preferably vary by organization. Rules addresssuch issues as whether users who are providing opinions must respond toall topics, are completely free to choose topics, or a mix of the twowherein certain topics must be replied to while replies to others areoptional. Exemplary rules will be set forth below in connection with theoperational description of preferred embodiments.

It is important to note that the division of databases discussed aboveis but one possible division. For example, one of the five databasesdiscussed above includes topics, issues and queries. Those of skill inthe art will recognize that these could also be separated into differentdatabases. Alternatively, one or more of the databases described abovecould be combined into a single database. It is not important whetherthe information in the databases described above resides in one or moreseparate files or on one or more separate data storage devices; rather,it is the logical interrelationships between the information that isimportant.

In operation, users of the continuous feedback system perform sevenmajor tasks. Not all tasks are performed by all categories of users. Theseven tasks are as follows:

1. Registration

2. Log-In

3. Prioritization of Topics and Issues

4. Provide Perceptions and Opinions

5. Evaluation of Responses

6. Generation of Reports

7. System Administration

Each of these tasks will be discussed in detail below. Rules applicableto each task in preferred embodiments will also be discussed.

Registration: Upon startup, a user is presented with a Login window 200as shown in FIG. 2. A user who has not yet registered clicks on the“here” hypertext 210, at which point the user is presented with theRegistration window 300. The Registration window 300 includes dialogboxes 310-360 in which a new user may enter the appropriate registrationinformation including first name, last name, email address, a user nameand a password. The following actions/responses and business rulesdefine this process in preferred embodiments:

System action—the system will present a user with a screen that allowshim or her to self-assign a user name and password

User activity—enter a user name and password

System response—the system will accept or reject a user name; if a nameis accepted it will store the name and password in the respondentdatabase

Rule—The system will either accept a user name if it matches a name on alist provided by the using organization or accept all user names forlater validation if there is no list provided by the using organization

Once the requested identification information has been entered, the useris presented with the Respondent Category Selection window 400 as shownin FIG. 4. The user may indicate, in one or more checkboxes 410, thecategories to which the user belongs. In some embodiments, the user isallowed to select whatever categories he or she desires. In suchembodiments, the user's selections may be compared to a list ofappropriate categories provided by leaders and modified to match thelist as required. In other embodiments, such a list may be providedbeforehand and used by the system to reject selections made by the user.Of course, if such a list is provided in advance, it is possible to havethe system automatically perform this step; however, there is some valuein having the user perform this process in any event as there is aneducational value to being exposed to the various categories. Asdiscussed above, checkboxes 410 are preferably provided for each entryin Table 1, regardless of its level in Table 1. Preferably, a user isautomatically entered in all higher level categories upon the selectionof a lower level category. Upon selection of appropriate categories, theRegistration process is complete.

The following actions/responses and business rules define this processin preferred embodiments:

System action—the system presents a user with a screen that allows himor her to record the user categories to which he or she belongs (seeFIG. 4)

User activity—the user will check all categories to which he or shebelongs

System response—the system will either accept or reject categories forthis user; if a category is accepted it is stored in the respondentdatabase with this user's data

Rule—accept categories for each user if they match a list provided foreach user by the using organization or accept categories for latervalidation if there is no list provided by the using organization; elsereject and require user to re-select categories.

LogIn: Registered users must log in each time the system is used. Uponaccessing the system, a user is presented with LogIn window 200 asdiscussed above. The user types the appropriate user name and passwordat dialog boxes 220 and 230 and clicks on the Login button 240. If theuser name and password information are correct, the user is presentedwith the Profile window 500 of FIG. 5 in one preferred embodiment. Thefollowing actions/responses and business rules define this process inpreferred embodiments:

System action—the system presents users with a screen that has a button510 that allows them to enter or modify their user categories (option 1)or select a single user category from their profile for this session(option 2) (see FIG. 5)

Option 1

User activity—the user will click the “button” if his or her usercategory data was not entered at registration or has changed

System response—the system provides the user with a screen that allowsthe user to enter user category data (same screen used duringregistration)

User activity—the user checks all categories to which they belong

System response—the system either accepts or rejects new usercategories; store data in respondent category database

Rules—the organization establishes rules on when and under whatcircumstances users may change their user profile (respondentcategories); the rules from the register function for accepting orrejecting user categories apply here also

Option 2

User activity—select the user category they are using for this session

System response—the system provides a main menu with options that areappropriate for the user category selected.

After the profile selection, the user may then select an activity whichhe wishes to perform. This is preferably accomplished through theActivity Selection window 600 of FIG. 6. This window 600 provides alisting 610 of activities. In preferred embodiments, only thoseactivities for which a user is authorized are presented in the listing610.

Prioritization of Topics and Issues: One of the most importantactivities is the prioritization of topics and issues. If the userselects the “Specify Hot Topics” activity from listing 610 (topicprioritization is performed by specifying a topic as hot in preferredembodiments), the user is presented with the Hot Topic Specificationwindow 700 of FIG. 7. This window contains a listing 710 of all topicsin Table 2. The user clicks on any topic in listing 710. Upon selectionof a topic, that topic is designated as a hot topic. Designation of atopic as a hot topic may have one or more consequences as specified inthe business rules database. Among the possible consequences are 1) thatthe topic may be displayed more prominently than non-hot topics, eitherby being marked by a special character or being displayed at the top ofthe list of available topics; and 2) that a user may be forced toprovide opinions for a designated hot topic before being allowed toprovide opinions on other topics. In preferred embodiments, a singleindividual is designated with the authorization to select and deselecthot topics—this individual may be the senior person in the organizationor someone designated by the senior person.

The following actions/responses and rules define the Hot TopicSpecification process in preferred embodiments:

User activity—the user selects Prioritize HOT Topics from listing.

System response—the system provides a screen with a list of topics forwhich user is authorized to mark as “hot topics”; obtain authorized listfrom authorization table in the business rule database

Rule—the senior person in the organization delegates authorization; thisdelegation is maintained in business rule database

User activity—the user selects new “hot topics” and de-select existing“hot topics” by highlighting each topic on the list

System response—the system will add and remove topics from the “hottopics” list

A user may also select “Prioritize Issues” from the listing 610. In thissituation, each topic is assigned a topic owner. The topic owner has thesole authority to prioritize issues (as listed in Table 2) in preferredembodiments. In preferred embodiments, issues are prioritized byassigning a percentage of queries to each issue in the topic.

The following actions/responses and business rules define this processin preferred embodiments:

User action—the user selects “Prioritize Issues” from listing 610.

System response—the system ensures the user is authorized to prioritizeissues; then provides a screen with a list of all issues in the databasefor authorized user to prioritize

Rule—only the topic owner may set the percentages of query statementsfor each issue in a topic

User activity—the user enters the percentage of query statements thatare to be selected from the database for each issue associated with atopic

Rule—the user may enter any percentage for an issue between 1% and 100%;however, the total for all issues that are selected for prioritizationmay not exceed 100%

System response—the system will store the data in the business ruledatabase; later, the system provides the appropriate percentages to thesoftware function that randomly selects the query statements from thedatabase for each topic for every respondent. Note: this percentage willbe used to ensure that whenever an issue is associated with a topic,that the appropriate percentage of queries related that issue arepresented to each respondent who selects that topic

Provide Perceptions and Opinions: Referring now back to FIG. 6, anotherimportant task a user may perform is to provide perceptions andopinions. If this activity is selected from listing 610 (recall that notall activities in listing 610 may be presented for all users), the useris presented with the Topic Selection window 800 of FIG. 8, which liststopics from which the user may select topics for which to provideopinions. Upon selection of topics, the user is presented with the Querywindow 900 of FIG. 9. The Query window 900 provides a plurality ofqueries 910. The user clicks on a button corresponding to a responsebased on a 7 point Likert scale for each query 910.

The following actions/responses and business rules define this processin preferred embodiments:

User activity—the respondent selects “voice your opinions” from listing610

System response—the system provides a list of topics that areappropriate for the respondent's user category chosen above—separatetopics listed for each of three roles (customer, performer andsupervisor) that a respondent might play

Rules—the system will mark all “hot topics” (selected by a senior personas discussed above) that are appropriate for this respondent's usercategory and place them at the top of the list of topics from which therespondent may choose

User activity—the respondent selects a topic for which he or she willprovide feedback

Rules—organizations establish if a respondent must select at least oneor more “hot topics” before selecting a topic that is not listed as a“hot topic”

System response—the system provides a set of query statements on a“point and click” response screen (see FIG. 9). Query statements arechosen random randomly from the issues—matching the percentages by issueif the topic owner has specified a percentage for one or more issues.The percentages are stored in the business rule database.

Rules—the number of query statements for each topic is determined by theorganization; they may change this as often as they deem desired.

User activity—the respondent chooses a response for each query statementfrom the 7 options that range from “strongly disagree” to “stronglyagree;” respondent may select “n/a” which stands for “not applicable.”After reviewing the responses the user may choose to respond to anothertopic, select an additional respondent category and repeat the processstarting with select topic, or log-off.

Rules—the organization determines how many topics a respondent mayprovide feedback on during each time period. This rule will apply to thenumber of topics for each role (respondent category) a user may belongto. That is, a user may respond to the chosen number of topics for eachcategory he or she belongs to.

Perform Evaluation: This is one of the most important functions that isperformed using the system. Upon selecting “perform evaluations” fromlisting 610, the user is presented with a Selection Window 1000 as shownin FIG. 10. The window 1000 includes a listing 1010 of all topics andissues in Table 2. The user may select any topic or issue which they areauthorized to evaluate. In preferred embodiments, all levels ofleadership/management are authorized to select topics and issues forevaluation. It is believed that one of the many benefits provided by thepresent invention is that a continuous feedback process will allow lowerlevel leaders and managers to quickly spot many problems as they ariseand take appropriate action before they become serious enough to attractthe attention of more senior leaders/managers.

If a topic is selected for analysis, a Summary window 1100 including asummary graph 1110 for that topic is presented as shown in FIG. 11. Thesummary graph 1110 shows all responses for all issues associated withthe selected issue (in this case, the General Achievement Testingissue). The responses are divided temporally between current responses(less than 15 days old), responses between 15 and 45 days old, andresponse greater than 45 days old. The selection of these time periodsis variable depending upon the needs of the organization.

If a more detailed analysis is provided, the user is then provided withthe Evaluation Options window 1200 of FIG. 12. The user may select toevaluate the responses by participant category or issues, and may choosewhether to evaluate all ratings, negative ratings only, positive ratingsonly, or a specific rating number from the Likert scale. The user mayalso choose the time period of the data to be reviewed (e.g., current,etc.).

If the user chooses to evaluate responses by participant category, theParticipant Category Evaluation window 1300 of FIG. 13 is displayed.This window 1300 includes a listing 1310 of responses (positive andnegative) for all issues associated with the topic by participantcategory (the lowest categories listed in Table 1). The total numbers ofquery responses for each category is shown. This technique allows anevaluator to determine whether satisfaction or dissatisfaction is moreprevalent among some groups rather than others.

If, on the other hand, the user chooses to analyze the data by issue,the Issue Evaluation window 1400 of FIG. 14 is displayed. Window 1400includes a listing, by each issue in the topic, of the responses fromall respondents (in FIG. 14, the user has chosen to analyze negativeresponses only by issue). This allows an evaluator to determine whethera specific issue if of particular concern for a given topic.

The user is also given the opportunity in window 1200 to specify trendanalysis. In this case, a Trend Analysis window 1500 as shown in FIG. 15is displayed to the user. This window 1500 shows the trend of theresponses (in this case, the percentages of positive responses) over thepreviously discussed time periods. Thus, for example, there is a trendtowards a perceived improvement in the effectiveness of generalachievement testing over the time periods indicated in FIG. 15. Trendanalysis helps leaders and managers quickly spot progress in addressingissues for issues applicable to a given topic.

In addition to the analyses described above, analysis for specificqueries is also possible as shown in the Query Analysis window 1600 ofFIG. 16. The window 1600 will include a listing 1610 of each query and atotal of each desired point on the Likert scale in preferredembodiments. A summary window such as the Trend Summary window 1700 ofFIG. 17 is also provided in preferred embodiments. In some embodiments,the system will automatically alert a user (preferably a user withleadership responsibility for a topic and/or issue, that a significanttrend has developed. Significance can be measured by percentages; thethreshold is preferably set by the user to be alerted. Thus, forexample, the system can automatically alert (e.g., by sending email orposting a message on the first window the user sees upon logging on tothe system) a leader to a drop of fifteen percent or more in thepercentage of users currently providing positive comments on the topicof general achievement testing, measure with respect to old data (>45days old) and/or newer data (14-45 days old).

The following actions/responses and business rules define this processin preferred embodiments:

User activity—the evaluator selects “perform evaluations” from listing610

System response—the system provides as screen with either a list oftopics and a list of issues the specific evaluator is authorized toevaluate; list will identify positive and negative trends for each topicand issue

Rule—the organization assigns each topic and issue to an “owner;” thisauthorization is stored in the business rule database. An owner mayauthorize additional users to evaluate the topics and issues they own;these authorizations are stored in the business rule database

User activity—the evaluator selects topic (options 1) or issue (option2) by either clicking on the topic/issue

Option 1

System response—the system provides a screen with a summary graph(histogram), reflecting the 7 potential Likert scale responses, thatshows the results of all responses on this topic; also provide a summaryof positive (the three blocks on the histogram with the highestratings—5,6,&7) and negative responses (the three blocks on thehistogram with the lowest ratings—1,2,&3) (see FIG. 11)

User activity—the evaluator reviews responses; if desired, evaluatorselects: detailed analysis (option 1a), review trend summary by doubleclicking on the “trend icon” at the end of the summary (option 1b),return to previous menu that allows user to select a new topic or issueto evaluate (option 1c), or print the selected information (Option 1d).

Option 1a

System response—the system provides the evaluator with a menu thatallows him or her to analyze the responses by: issue, respondentcategory, time period in which responses were received, or the specificresponse level on the 7 point scale (see FIG. 12)

User activity (option 1a)—the evaluator selects: a) issue or respondentuser category; b) type of ratings (all ratings, all positive ratings,all negative ratings, or specific ratings), and c) age of data desired(one of three periods for which data is stored)

System response—the system provides a screen that shows the percentageof responses for each issue that falls in the response level chosen (seeFIGS. 13, 14 and 15)

User activity—the evaluator reviews the information; evaluator can:print the screen (options 1a1), double click on any bar in the histogramto show specific query responses that provide that the data for that bar(option 1a2), or return to the previous menu for choosing the type ofdetailed analysis (option 1a3).

Option 1a-1

System response—the system captures the page and sends it to the defaultprinter

Option 1a-2

System response—the system provides a screen that lists the specificquery statements that were answered to provide the data in the bar thatis double clicked; the list provides the total number of times eachquery statement was answered at each level on the 7 point Likert scale

Option 1a-3

System Response—the system returns to the detailed analysis menu

Option 1b

System Response—the system provides a summary of the positive ornegative trend data (see FIG. 17)

User activity—the evaluator can return to the list of “topics toevaluate” by clicking on the “close window” option, or the evaluator may“view the summary graph” by clicking on that option (this takes the userto the same graph described in option 1a above)

Option 1c

System Response—the system will return the user to the menu that allowshim or her to select additional topics or issues to evaluate

Option 1d

System response—the system captures the page and sends it to the defaultprinter

Option 2

System response—this response and all subsequent system actions andresponses are identical to those for Option 1 with the topics and issuesinterchanged.

User activity—this activity and subsequent user activities are identicalto those for Option 1 with the topics and issues interchanged.

Rules—the rules are the same as for Option 1 with the topics and issuesinterchanged.

Generation of Reports: The system also provides the ability to generatecustomized reports. In preferred embodiments, customized reportscomprise a selection of the analyses discussed above. The reports may begenerated automatically at fixed intervals and/or may be generated uponrequest by an evaluator.

The following actions/responses and business rules define this processin preferred embodiments:

User activity—the user selects Generate Standard Reports from the mainmenu

System response—the system provides the user with a menu that containsall the customized reports available to the user

Rules—the organization determines which customized reports they need andwho has the authority to generate each report; this data is stored inthe business rule database.

User activity—selects the report desired

System response—the system: 1) extracts responses for all queries thatare relevant to the requested report, 2) stores relevant queries foreach report in a table for that report in the response database, 3)performs statistical analysis, and 4) inserts the results into thereport template.

System response—the system will provide an options menu on what to dowith the report

User activity—the user selects one or more from the options menu: printthe report (option 1), store the report as a Microsoft Word document(option 2), email the report (option 3), and/or insert the report inanother document (option 4).

Option 1

System response—the system captures the document and sends it to thedefault printer

Option 2

System response—the system asks the user to specify a storage location

User Activity—the user selects the storage device, file and folder asappropriate

Option 3

System response—the system checks to see if there is an Internetconnection; if there is none it informs the user and allows him or herto establish a connection; if there is one the system provides thecomputers default email menu

User Activity—the user provides email address and any message ifdesired. The user then attaches the report; if the report has not beenstored the user must do this before attaching it to the email. The userthen clicks on the email send button.

Option 4

System response—the system asks the user for the document that thereport should be inserted into by specifying the file name and path

User activity—the user specifies the file name and path

System response—the system ensures that the document can accept aMicrosoft Word document; if so it asks the user to specify the page inthe document where the report is to be inserted; the system inserts thereport in the documents and asks the user to “save” the document; if thedocument cannot accept a Microsoft Word document the system informs theuser and allows him or her to return to the previous menu.

System Administration: This task comprises tasks associated with addingand removing queries, users, topics, issues, etc. to and from thesystem. The mechanics of these tasks will vary depending upon thespecific application and will not be discussed in detail herein.

As discussed above, the system is intended to be used continuously. Thatis, users provide input in the form of responses to queries many times.For example, students, teachers and other users in a school systemembodiment may answer queries daily, weekly, bi-weekly, or at some otherperiodic rate. In preferred embodiments, each time a user logs on andindicates that they wish to provide feedback, a new set of queries isselected from the database for each topic selected by the user. Inpreferred embodiments, between six and twelve queries are presented foreach topic. These queries are spread among the issues applicable to thetopic in the proportions selected by the topic owner. In someembodiments, queries are selected such that there is at least one queryfor each issue applicable to the topic. Because the database of queriesfor each topic is large (e.g., 80-400 per topic), the user will bepresented with new questions each time he or she selects a given topic.Thus, although the user is providing opinions continuously, the userwill not become disinterested by seeing the same questions each timethey log on.

Obviously, numerous modifications and variations of the presentinvention are possible in light of the above teachings. It is thereforeto be understood that within the scope of the appended claims, theinvention may be practiced otherwise than as specifically describedherein.

What is claimed is:
 1. A method for allowing management to gatherqualitative information from a plurality of employees of an organizationover a communications network, comprising the steps of: identifying, andstoring in a database connected to the communications network, aplurality of categories of employees of an organization from whominformation is desired, and topics and issues for which information isdesired; accessing said database to formulate queries applicable to eachcombination of one of said plurality of categories, topics and issues;assigning each of the plurality of employees of an organization to oneof said plurality of categories; receiving a selection of a desiredtopic from each of the plurality of employees; presenting, via thecommunications network, a portion of the queries to each of theplurality of employees based upon said assigned categories, wherein saidportion of the queries is relevant to said desired topic selected byeach of the plurality of employees; and gathering responses to saidportion of the queries from each of the plurality of employees, saidresponses being gathered on a Likert scale and stored in said database;wherein management can access said database to identify trends withrespect to perceptions and opinions of the plurality of employees onspecific topics and issues.
 2. The method of claim 1, further comprisingthe step of: repeating each of said presenting and gathering steps atleast once, wherein different portions of said queries are presented toeach of the plurality of employees upon each repetition of saidpresenting step.
 3. The method of claim 1, further comprising the stepof: displaying all responses for a given topic in a format in which afirst axis represents a value on said Likert scale and a second axisrepresents a number of responses.
 4. The method of claim 3, wherein saidformat is a histogram.
 5. The method of claim 3, wherein said format isa bar graph.
 6. The method of claim 3, wherein responses are segregatedtemporally.
 7. The method of claim 3, wherein each point on said firstaxis corresponds to a single point on said Likert scale.
 8. The methodof claim 3, wherein each point on said first axis corresponds to acombination of points on said Likert scale.
 9. The method of claim 1,further comprising the step of: displaying all responses for a giventopic in a format in which a first axis represents a category ofemployees and a second axis represents a number of responses in aportion of said Likert scale.
 10. The method of claim 9, wherein saidportion is a positive portion.
 11. The method of claim 9, wherein saidportion is a negative portion.
 12. The method of claim 1, wherein saidLikert scale is a seven point Likert scale.
 13. The method of claim 1,wherein all of said plurality of issues are applicable to at least twoof said plurality of topics.
 14. The method of claim 1, wherein one ofthe plurality of employees is assigned to at least two of said pluralityof categories.
 15. The method of claim 1, wherein one of the pluralityof employees is assigned to one of said plurality of categoriesaccording to a selection received from said one of the plurality ofemployees.
 16. The method of claim 1, wherein one of the plurality ofemployees is assigned to one of said plurality of categories based on apredetermined selection.
 17. The method of claim 1, wherein a portion ofthe queries presented to one of the plurality of employees correspond toa topic selected by a topic owner.
 18. The method of claim 1, furthercomprising the steps of: accepting an input indicative of a topic forwhich responses are to be displayed; and displaying responses from allof the plurality of employees who responded to queries related to saidtopic.
 19. The method of claim 1, further comprising the steps of:accepting an input indicative of an issue for which responses are to bedisplayed; and displaying responses from all of the plurality ofemployees who responded to queries for issues related to said topic. 20.The method of claim 1, further comprising the steps of: accepting adesignation of one of the plurality of employees as an owner of at leastone topic; accepting a specification by said topic owner of a number ofqueries for each issue comprising said at least one topic; presentingsaid number of queries for each issue comprising said at least onetopic, wherein said number of queries is based upon said specificationby said topic owner.
 21. The method of claim 1, further comprising thesteps of: accepting a first input indicative of one of said plurality ofcategories for which responses are to be displayed; accepting a secondinput indicative of a selection from the group consisting of topics andissues; and displaying the number of responses corresponding to eachpossible response from said Likert scale for each query responded to bythe plurality of employees corresponding to the selections made in saidaccepting steps.
 22. The method of claim 1, further comprising the stepsof: identifying, and storing in the database, at least one category ofnon-employees from whom information is desired; accessing said databaseto formulate a second set of queries applicable to each combination ofthe at least one category of non-employees, the topics, and the issues;assessing at least one non-employee to the at least one category ofnon-employees; receiving a selection of a desired topic from the atleast one non-employee; and presenting, via the communication method, asecond portion of the second set of queries to the at least onenon-employee, wherein the second portion is relevant to the desiredtopic selected by the at least one non-employee; wherein the gatheringstep further includes gathering responses to the second portion by thenon-employee.
 23. The method of claim 22, wherein the non-employee is acustomer.
 24. A system for allowing management to gather qualitativeinformation from a plurality of employees of an organization,comprising: a database server; a plurality of end user computers; and acommunication network connected to said database server and saidplurality of end user computers; wherein said database server includes adatabase comprising: a plurality of categories of employees of anorganization; a plurality of employees wherein each of said plurality ofemployees belongs to at least one of said plurality of categories; aplurality of topics; a plurality of issues; and a plurality of querieswherein each of said plurality of queries is applicable to at least onecombination of one of said plurality of categories, topics and issues;wherein said database server is configured to perform the steps of:accepting a selection of a desired topic from one of said plurality ofemployees; selecting a portion of said plurality of queries forpresentation to said one of said plurality of employees based upon thecategory of said one of said plurality of employees, wherein saidportion of said plurality of queries are relevant to said desired topic;presenting said portion of said plurality of queries to said one of saidplurality of employees; and gathering responses to said portion of saidplurality of queries, said responses being gathered on a Likert scaleand stored in said database; and wherein management can access saiddatabase to identify trends with respect to perceptions and opinions ofsaid plurality of employees on specific topics and issues.
 25. Thesystem of claim 24, wherein said database server is further configuredto perform the step of: repeating each of said accepting, selecting,presenting and gathering steps for each of said plurality of employees.26. The system of claim 24, wherein all issues are applicable to atleast two topics.
 27. The system of claim 24, wherein said Likert scaleis a seven point Likert scale.
 28. The system of claim 24, wherein saiddatabase server is further configured to perform the steps of: acceptinga designation of said one of said plurality of employees as an owner ofat least one topic; accepting a specification by said topic owner of anumber of queries for each issue comprising said at least one topic; andpresenting said number of queries for each issue comprising said atleast one topic, wherein said number of queries presented is based uponsaid specification by said topic owner.
 29. The system of claim 24,wherein the database further comprises at least one second category ofnon-employees from whom information is desired and a second set ofqueries applicable to at least one combination of an issue, a topic, andthe at least one second category, wherein the database server is furtherconfigured to perform the steps of: accepting a selection of a desiredtopic from a non-employee assigned to the at least one second category;selecting a second portion of the second set of queries for presentationto the non-employee based upon the at least one second category, whereinthe second portion is relevant to the desired topic selected by thenon-employee; presenting the second portion of the second set of queriesto the non-employee; and gathering responses to the second portion ofthe second set of queries.
 30. The system of claim 29, wherein thenon-employee is a customer.